Statistical outlier? Retrospective analysis using routine hospital data to identify gynaecologists' performance -- Harley et al.

Was Rodney Ledward a statistical outlier? Retrospective analysis using routine hospital data to identify gynaecologists' performance -- Harley et al. 330 (7497): 929 -- BMJ
Useful methods for monitoring performance
Although scanning methods14 such as ours will never have complete diagnostic certainty, they could be used to reliably identify signals from noise,13 which need to be systematically and sensitively examined, perhaps confidentially, by peers.21 Although our methods urgently need to be evaluated prospectively, organisations engaged in this type of performance monitoring, including the National Patient Safety Agency, the Healthcare Commission, the General Medical Council, the NHS Litigation Authority, and the National Clinical Assessment Authority may find our methods of interest. Nevertheless, although the ability to identify poorly performing clinicians after the event has its uses, prevention is preferable; but this presents an altogether different challenge—one that seeks to engineer the safety of patients into the process of care by

Posted as supplied by author.

Appendix

Robust Multivariate Outlier Detection using the Mahalanobis Distance

We first undertook a robust principal component analysis, for each year (and all years together) which showed that the seven indicator variables were orthogonal (results not shown). We then carried out a robust multivariate outlier detection analysis, based on the computation of a robust Mahalanobis Distance (MD)[9] for each consultant in each year.

The MD is in essence a weighted (by the sample robust variance-covariance matrix) Euclidean distance. The MD is measured from the centroid of the indicator variable-space, so that a consultant with average values for each variable will have a MD of zero, and this represents the origin. Consultants who are furthest away from the centroid, irrespective of direction will have relatively larger MDs. We used the Stahel-Donoho robust multivariate estimator to compute our MDs (Maronna RA, Yohai VJ. The behaviour of the Stahel-Donoho robust multivariate estimator. J Am Stat Assoc 1995;90:330-41).

For each consultant, we derived approximate 95% intervals of uncertainty using simulation. Since each variable is based on its own sample size (n), which varies with each consultant, we attempted to account for this by taking repeated random samples from an underlying binomial distribution (based on n and p, for the six proportion based variables) and a normal distribution (for the mean length of stay variable). Exploratory analyses indicated that these were reasonable underlying sampling distributions. So for each consultant we produced 1000 randomly simulated values for each of seven variables, and then computed a robust MD in the manner described above from these simulated data sets. From these MDs, we used the 0.95 and 0.05 centiles to estimate the approximate 95% intervals of uncertainty around each consultant's MD.

The Ö MD distance is known to approximately follow a Ö χ2 distribution with k degrees of freedom (k being equal to the number of variables, 7 in our case)[9] with mean equal to Ö k.[9] Where a consultant's approximate 95% intervals of uncertainty were above the mean (Ö k=2.65), we deemed this consultant to be an outlier. In the presentation of results in Figure 1, we chose to express the Ö MD on the loge scale to aid visualization of the plots.

Multi-level modelling

We also investigated the change in MD over the five years, by taking the transformation loge(Ö MD) using a multi-level modeling approach with the repeated measurements (MD) nested within consultants. The reason for transforming the Ö MD to the loge scale was to avoid heteroscedasticity seen on visual examination of the residuals. The log likelihood of the model with Ö MD -857 versus -238 when using the transformed, loge(Ö MD). We used the standardised residual output from this model to identify outliers beyond +/- 2 standard deviations (Figure 2).